Method for interactive control of a computer application

ABSTRACT

The present invention relates to a system, a method and a related device for controlling an interactive computer application, where the system comprises an application executing part that is adapted to execute the interactive computer application. The system of the present invention the system further comprises a sensoring part that is adapted to detect at least one of presence and movements of a person in a capture area of the sensoring part. The application executing part further is able to switch the application from an idle mode into a passive/control discovery application mode at first detection of a person in the capture area of the sensoring part and in that the application executing part additionally is adapted to switch the application from the passive discovery mode into an interaction mode if the presence of the person is detected for a predefined period of time; and in that the application executing part is adapted to interpret movements of the person as control commands for/of the application if the application is in the interaction mode.

The present invention relates to method, a related system and relateddevices for interactive control of a computer application.

Such a method and a system for control of an interactive computerapplication where the interactive application running on a computersystem is well known in the art. Such an interactive computerapplication initially runs in an idle state if no person gives anexplicit start command, where the explicit start command for instancemay be a button push at the computer system or a touch at theapplication interface of such interactive computer application.

A camera based interactive application systems could be combined withwall or floor projections. However, currently, they are only used forpassive interactivity, meaning that if a pedestrian enters the capturearea of the camera something happens which cannot be directly influencedby the person. It is mostly based on a kind of motion detection. Realinteraction via camera demands the detection of natural and intuitivehuman gestures.

Such solution still requires the explicit starting of an application byany potential user pushing a button, although such potential userparticipant may even not be aware of existence of the interactivecomputer application and if aware will not be aware of what action isrequired to start.

Problem of this art is how to start controlling a computer applicationwithout explicit acknowledgment of a potential participant or user ofthe application to start the application as a potential user is notalways aware of the existence of the computer application and itspossibilities.

An objective of the present invention is to provide an applicationcontrol method and system and related computing device of the aboveknown type but wherein the control can be initiated in a more flexibleand intuitive manner.

According to the present invention, this object is achieved by thesystem according to claim 1, the method according to claim 8, a relateddevice as described in claim 9.

Indeed, according to the invention, this object is achieved due to thefact that by first detecting the presence of a person in the capturearea of a presence and motion sensoring part, the interactive computerapplication is switched from an initial idle mode into a second passivecontrol discovery application mode and subsequently, if the presence ofthis person still is detected during a certain predefined period oftime, the application is switched from the second passive discovery modeinto a third interaction mode wherein gestures and movements of theperson whose presence is detected are interpreted and accepted ascontrol commands of the interactive application, where the fact that aperson stays during a certain predefined period of time within thecapture area of the sensoring part is accepted as an interest forinteracting with the application consequentially serving as aconfirmation for interacting with the interactive computer application.

A further characteristic feature of the present invention is describedin claim 2.

The sensoring part comprises a camera that is adapted to detect presenceand movements of a person within the capture area of said camera.

A further characteristic feature of the present invention is describedin claim 3.

The sensoring part (SP) comprises a depth camera to detect presence andmovements of a person in said capture area of said camera. Such depthcamera could be n infrared camera. Such depth camera based on infraredadditionally is able to perform its functions in dark areas independentof incidence of light.

Still a further characteristic feature of the present invention isdescribed in claim 4 and claim 10.

The interactive computer application is an interactive advertisementapplication and the system further may or may not comprise an interfacebeing an advertisement billboard.

Still a further characteristic feature of the present invention isdescribed in claim 5 and claim 11.

The interactive computer application is an interactive gamingapplication where the interactive application may comprise an interfacebeing a gaming billboard. Alternatively the interface may be anydisplay.

Another further characteristic feature of the present invention isdescribed in claim 6 and claim 12.

The interactive computer application is an interactive multimediarendering application where the interactive application may comprise aninterface being a gaming billboard. Alternatively the interface may beany display.

Still a further characteristic feature of the present invention isdescribed in claim 7 and claim 13.

The interactive computer applications being in said passive controldiscovery application mode after a first detection of a person in thecapture area of the sensoring part displays said person on coupleddisplay. The interactive application for instance may show an avatar ofthe player at a coupled display or billboard. Hence the detected personwill see an introduction to real interactivity. At presentation of suchavatar the person will be aware of some kind of influence or control onthe shown content.

It is to be noticed that the term ‘comprising’, used in the claims,should not be interpreted as being restricted to the means listedthereafter. Thus, the scope of the expression ‘a device comprising meansA and B’ should not be limited to devices consisting only of componentsA and B. It means that with respect to the present invention, the onlyrelevant components of the device are A and B.

Similarly, it is to be noticed that the term ‘coupled’, also used in theclaims, should not be interpreted as being restricted to directconnections only. Thus, the scope of the expression ‘a device A coupledto a device B’ should not be limited to devices or systems wherein anoutput of device A is directly connected to an input of device B. Itmeans that there exists a path between an output of A and an input of Bwhich may be a path including other devices or means.

The above and other objects and features of the invention will becomemore apparent and the invention itself will be best understood byreferring to the following description of an embodiment taken inconjunction with the accompanying drawings wherein:

FIG. 1 represents an overview of the space wherein the present inventionmobile device wherein the application switching method according thepresent invention is executed locally.

FIG. 2 represents a system including a computing device CD and asensoring part wherein the controlling of an interactive computerapplication according to the present invention is executed.

In the following paragraphs, referring to the drawings, animplementation of the method for controlling an interactive computerapplication, a related system and related devices according to thepresent invention is described.

In the first paragraph of this description the main functional parts ofthe computing device CD as presented in FIG. 2 are described. In thesecond paragraph, all connections between the before mentioned elementsand described parts are defined. In the succeeding paragraph the actualexecution of the application switching method for session establishmentis described.

First essential element of the present invention is a sensoring part SPthat is adapted to detect at least one of presence and movements of aperson in a capture area of the sensoring part SP. This sensoring partmay be a commonly known camera or be a depth camera, like an infraredcamera.

A characteristic of such sensoring part SP is that it has a certaincapturing area, being the limited area wherein the sensoring part isable to detect presence of a person and movements of such person. Thiscapturing angle is amongst others limited due to the angle of the cameralens. The capture area in FIG. 1 is denoted with CA and the area outsidethe capture area as INCA.

A further relevant element of the present invention is a computingdevice that first comprises an application executing part PP that isfirst is adapted to execute the interactive computer application. Theapplication executing part PP further is adapted to switch theapplication from an idle mode into a passive control discoveryapplication mode at first detection of a person in the capture area ofthe sensoring part SP, i.e. a camera. The application executing part PPadditionally is adapted to switch the interactive application from thepassive control discovery application mode into an interaction mode ofthe interactive computer application if the presence of the person isdetected during a predefined period of time from the initial presencedetection. The interactive computer application is able to, by means ofthe application executing part PP, to interpret movements of said personas control commands of the application only if the application is in theinteraction mode. In case the interactive computer application is inidle mode the application does not offer any interactivity, but in caseof a multimedia application, gaming application or advertisementapplication provides with a basic and standard presentation runningrepeating continuously.

In the subsequent case that the interactive computer application is inpassive discovery mode the application does still not offer anyinteractivity, but in case of a multimedia application, gamingapplication or advertisement application provides with a basic andstandard presentation inviting a person who is detected to participatein the interactive computer application, e.g. being the multimedia show,the game or the advertisement.

The computing device CD further comprises a Reception part that isadapted to receive a signal from the sensoring part where the signal isa signal including presence and/or movement information on a detectedperson being located within the capturing area. In case the sensoringpart is a camera the signal are for instance the video signal recordedby the camera.

The computing device further comprises a Device interfacing part DIPthat is adapted to interface with a user interfacing part UIP e.g. beinga display or a billboard.

The user interfacing part UIP is the user interface of the system onwhich information of the interactive computer application is renderede.g. being the multimedia rendering, the game rendering or theadvertisement rendering.

The reception part RP of the computing device CD has an input-terminalthat is at the same time an input terminal of the computing device CD.The reception part RP further is coupled to application execution partAEP that in turn further is coupled to the Device interfacing part DIP.The Device interfacing part DIP further has an output-terminal that isat the same time an output-terminal of the computing device CD.

In order to explain the execution of the present invention it is assumedthat a certain pedestrian walks in direction of the system of thepresent invention, for controlling an interactive computer and aassociated billboard application assuming the pedestrian is outside thecapture area of the camera see FIG. 1. As in the current situation noperson is detected by means of the camera, as the person still islocated outside the capture area, the area being denoted with INCA, theinteractive computer application is in idle mode and hence theapplication does not offer any interactivity. In case the interactivecomputer application is a multimedia application, gaming application oradvertisement application, the interactive application provides with abasic and standard presentation running repeating continuously.

One possible scenario is an interactive gaming billboard. This billboardhas three states which states differ in the kind of interactivity. In astate in which nobody is detected within the captured area of the cameraa standard advertising loop is running on the billboard. If somebodyappears in this area the camera will find, track and follow the person.Now the billboard will react in a passive manner to show the person thatthere is interactivity possible and he can be part of it.

Subsequently, the pedestrian approaches the billboard coupled to thesystem of the present invention and consequently enters the capture areaCA of the coupled sensoring part i.e. the camera. The sensoring partdetects the presence of the pedestrian in the capture area CA, i.e. thecamera captures the presence of the pedestrian and sends the images viathe receiving part RP of the computing device CD to the applicationexecuting part PP that subsequently switches the application from theidle mode into the passive control discovery application mode at thefirst detection of a person e.g. by detecting the face or the skeletonof the person, in the capture area of the sensoring part.

The interactive computer application, being in passive control discoveryapplication mode, still does not offer any interactivity, apart fromthat the interactive application for instance may show an avatar of theplayer at a coupled display or billboard. Hence the detected person willsee an introduction to real interactivity. At presentation of suchavatar the person will be aware of some kind of influence or control onthe shown content.

In case of a multimedia application, a gaming application oradvertisement application provides with a basic and standardpresentation inviting the pedestrian being in the capture area CA toparticipate in the interactive computer application, e.g. being themultimedia show, the game or the advertisement. In this passive statethe avatar of the user is presented at the billboard and therefore thereis a feedback of the interactivity for the tracked person. Currently hecan move but there is not an interaction with the advertisement. Inaddition, the interactive application then may use dialogues like “heyyou! Would you like to play?” are displayed to get more attention of theperson in the capture area and to persuade him to participate in theapplication e.g. the multimedia application game or advertisementapplication. Such dialogues may be applied in order to let such personknow that he as a person is part of the advertisement. The dialogue isdirect communication which is more personal than a manual or aguideline.

In case the pedestrian subsequently is interested for some reason tointeract with the application and stays within the capture area in theobserved area, for a specific time period which is detected e.g. byfinding the face or skeleton of the person back in a sequence of framescaptured by the camera by means of the application executing part PP andswitches the application from the passive control discovery applicationmode into the interaction mode if the presence of the person is detectedfor a predefined period of time; e.g. by finding the face of the personback in a sequence of frames captured by the camera.

After switching the interactive computer application into theinteraction mode, the application executing part PP interprets movementsof the person as control commands of the application.

In this interactive mode of the computer application, all movements ofthe user are captured, analyzed and subsequently interpreted as controlsignals of the application. In the interactive state the applicatione.g. a game will be introduced and starts automatically. Now the usercan move and play, control the game with body movements. At the end ofthe game the result is visible together with a QR-Code as a give-away.When the user rests in the capture area the game will start again.

Now the person has control over the advertising. With this control agame could be played without any gadgets only by human gestures. In thisexample the gesture will be lateral body movements.

A camera recognizes people paying attention to the advertising solution.As remarkable feature frontal faces are used since interested personsface the application directly.

A camera recognizes people paying attention to the advertising solution.As remarkable feature also skeletons of people are used to guarantee astable system independent of incidence of light and in interactive modeit allows interaction without frontal faces.

When a person is detected he is focused and tracked until this personleaves the observed area, i.e. the capture area where after theapplication switches back into the idle mode of the interactiveapplication.

Focusing is done with a time based state machine to generate reliableresults.

Tracking results like position and size changes can be used by theadvertising application to customize the displayed content.

If no person is detected during a specified period of time the systemswitches to the idle mode state.

In both candidate and locked state the algorithm changes to initialstate if the result could not be detected again and an according timeoutexpires e.g. when the tracked user left the area.

Beside the gaming idea also an exploring mode could be an opportunity inwhich gestures like arm wipes can be used similar to the iPhone screennavigation to change between images or use virtual objects.

Also augmented reality or 3D effects are a realistic extension to anykind of advertising campaign for the interactive gaming billboard.

It is to be noted that the sensoring part of in the present inventioncould be implemented by any other device able to detect presence andmotion of a person.

A final remark is that embodiments of the present invention aredescribed above in terms of functional blocks. From the functionaldescription of these blocks, given above, it will be apparent for aperson skilled in the art of designing electronic devices howembodiments of these blocks can be manufactured with well-knownelectronic components. A detailed architecture of the contents of thefunctional blocks hence is not given.

While the principles of the invention have been described above inconnection with specific apparatus, it is to be clearly understood thatthis description is merely made by way of example and not as alimitation on the scope of the invention, as defined in the appendedclaims.

1. System for controlling an interactive computer application, saidsystem comprising an application executing part, for executing saidinteractive computer application said system further comprising: asensoring part, adapted to detect at least one of presence and movementsof a person in a capture area of said sensoring part; and saidapplication executing part, further being adapted to switch saidapplication from an idle mode into a passive control discoveryapplication mode at first detection of a person in said capture area ofsaid sensoring part; and in that said application executing partadditionally is adapted to switch said application from said passivediscovery mode into an interaction mode if said presence of said personis detected for a predefined period of time; and in that saidapplication executing part is adapted to interpret movements of saidperson as control commands for/of said application if said applicationis in said interaction mode.
 2. System for controlling an interactivecomputer application according to claim 1, wherein said sensoring partcomprises a camera adapted to detect presence and movements of a personin said capture area of said camera.
 3. System for controlling aninteractive computer application according to claim 1, wherein saidsensoring part comprises a depth camera to detect presence and movementsof a person in said capture area of said camera.
 4. System forcontrolling an interactive computer application according to claim 1,wherein said interactive computer application is an interactiveadvertisement application.
 5. System for controlling an interactivecomputer application according to claim 1, wherein said interactivecomputer application is an interactive gaming application.
 6. System forcontrolling an interactive computer application according to claim 1,wherein said interactive computer application is an interactivemultimedia rendering application.
 7. System for controlling aninteractive computer application according to claim 1, wherein saidinteractive computer applications being in said passive controldiscovery application mode after first detection of a person in saidcapture area of said sensoring part displays said person on coupleddisplay.
 8. Method for controlling a computer application said computerapplication, said computer application being executed by an applicationexecuting part, said method comprising: detecting presence of a personin a camera capture area of a sensor; switching said application from anidle mode into a passive control discovery application mode at firstdetection of a person in said capture area of said censoring part; andswitching said application from said passive discovery mode into aninteraction mode if said presence of said person is detected for apredefined period of time; and interpreting detected movements of saidperson as control commands of said application if said application is insaid interaction mode.
 9. Computing Device, for use in a systemaccording to claim 1, wherein said device comprises an applicationexecuting part, for executing said interactive computer application saiddevice further comprising: a sensoring reception part, adapted toreceive at least one of presence and movements of a person in a capturearea detected by said sensoring part; and said application executingpart, further being adapted to switch said application from an idle modeinto a passive control discovery application mode at first detection ofa person in said capture area of said sensoring part; and in that saidapplication executing part additionally is adapted to switch saidapplication from said passive discovery mode into an interaction mode ifsaid presence of said person is detected for a predefined period oftime; and in that said application executing part is adapted tointerpret movements of said person as control commands for/of saidapplication if said application is in said interaction mode. 10.Computing Device for controlling an interactive computer applicationaccording to claim 9, wherein said interactive computer application isan interactive advertisement application.
 11. Computing Device forcontrolling an interactive computer application according to claim 9,wherein said interactive computer application is an interactive gamingapplication.
 12. Computing Device for controlling an interactivecomputer application according to claim 9, wherein said interactivecomputer application is an interactive multimedia rendering application.13. Computing Device for controlling an interactive computer applicationaccording to claim 9, wherein said interactive computer applicationbeing in said passive control discovery application mode after firstdetection of a person in said capture area of said sensoring partdisplays said person on a coupled display.